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Data  Fusion  is  Essentiai  to 
interoperability  and  Information  Utility* 


•  Expect  reduced  expenditures  for  expensive  sensor  piatforms: 

-  Reduced  coverage 

-  Fewer  specialized  missions 

-  Less  redundancy  /  Fewer  alternatives 

•  Therefore,  increased  emphasis  wiii  be  piaced  on  processing,  to 
do  more  with  less 

•  Fiow,  access  and  use  of  information 

-  Communications  bandwidth  and  connectivity 

-  Data  Correlation  and  Fusion 

-  Collection  Management  &  Battle  Management 


*  Diane  Roark,  Senior  Staff  Member,  House  Permanent  Seiect  Committee  on 
Inteitigence;  presented  at  Sl/l/C  Combat  Info/lntei  Correiation  Symp.,  Jan  96) 
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Sensor  Fusion  Benefits 


•  Combine  Multi-Source/Multi-Discipline  Information 

-  Target  refinement:  location/track,  ID 

-  Cross-domain  imaging  (e.g.  E-O-hSAR) 

-  Force  structure  assessment 

-  Own  force  vulnerability  assessment 

-  Supports  Planning/Plan  Execution/Re-planning 

•  Maintain  Track  Continuity:  Correlate  Time-Separated  Observations 

-  Intermittent  sensor  passes 

-  Terrain  masking  and  countermeasures 

•  Cross-Sensor  Cueing 

-  More  efficient  search 

-  Enhanced  detection 
(cued  dwells;  reduced  threshold) 

-  Reduced  requirements  on  individual 
sensors  (sensitivity,  coverage,  accuracy) 

-  LO  techniques  (passive  cueing  of  radar;  bistatic  sensing) 
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General  Deficiencies  in 
Data  Fusion  Systems 


EFFECTIVENESS: 

-  Performance:  Lack  of  timely,  accurate  target  &  situation  awareness 

-  Focus:  Information  not  tailored  to  decision-maker’s  needs 

-  User  Confidence:  Can’t  assess  information  quality 

-  Interoperability:  Legacy  systems  can’t  talk  to  one  another 

-  Data  Exploitation:  Reported  data  doesn’t  include  some  types  of 
useful  data 

AFFORDABILITY: 

-  Every  new  system  is  designed  from  scratch 
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Problem  Analysis 


Failure  to  use  available  data  and  designs 
Lack  of 

-  Standard  methods  for  specifying  data  fusion  problems 

-  Objective  performance  metrics 

-  Generally  accepted  comparative  analyses  of  alternative  techniques 

-  Standard  multi-spectral/multi-discipline  target/collection  models 

Diverse  data  fusion  systems  can  be  represented  under  a  common 
paradigm  but  it  requires: 

-  Requirements  &  MOPs 

-  Target/ Collection  Models 

-  Architectures  &  Functions 

-  Data  Sets 

Use  of  such  a  paradigm  would  facilitate 

-  System  Acquisition  -  Operation/  Interoperability 

-  Adaptation/Re-use  -  Evaluation 
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Sensor/Data  Fusion  Working  Group 

Famiiy  Tree 


DDRE 


IRIS 


ASD/C^I 


JDL 

(Joint  Directors  of  Laboratories) 

TPIST 

(Tech  Panei  on  information  Systems  Technoiogy) 


SDF  Specialty  Group 


SDFWG  ^ - - 

(Sensor/Data  Fusion  Working  Group) 

Group  DFG 

(Data  Fusion  Grom 


Sensor/Data  Fusion  Symposia 

Otherinformation  Exchange  Functions  Standards  Analysis  Special  Purpose  etc. 

Workshops 

All-Pervasive  Nature  of  Data  Fusion  | 
Requires  a  Unified  Organizational  Approach  I 
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Sensor/Data  Fusion  Working  Group 


To  deveiop  a  successfui  Fusion  process  (automated  or  manuai)  a  good 
understanding  of  the  foiiowing  is  required: 


1.  Physics  of  the  Sensor/Coiiector  AND  the  Phenomena  (NSSF) 

2.  Data  Fusion  Processes  (DFG) 

3.  Warfare  Mission  Area 

4.  Customer/User 

•  in  the  current  Era  of  C4iSR  and  emphasis  and  need  for  integration,  it  is 
past  time  that  we,  as  a  community,  started  working  on  these  issues. 
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Key  Data  Fusion  Problems^^^oi 


•  “Easy”  Problems:  Solutions  Known  ^  Coordination,  Funding,  Engineering 

-  Stovepipes 

C3  Architecture 

-  Interoperability  &  Data  Latency 

C3  Arch,  Interfaces  &  Processing 

-  Spatio-Temporal  Alignment 

C3,  Registration,  Calibration 

-  Integration  with  Collection  Management 

C3  Arch,  Process  Control 

-  Confidence  Normalization 

Source  Modeling,  Reporting 

-  High  Density,  Dynamic  Environments 

Hyp  Generation,  Hyp  Selection 

•  Difficult  Problems:  Fundamental  Research  Needed  ^  Science 

-  Predictable  Performance 

Modeling  &  Simulation 

-  Highly  Ambiguous,  Noise-Like  Indicators 

Context  Sensitive  HG,  HITL 

-  “Subtly-Related”  Indicators 

Adaptive  Fusion  Trees,  HITL 

-  Low  Confidence  Models 

Hyp  Generation,  Hyp  Selection, 
Dynamic  Databases,  HITL 

HITL  =  Human-in-the-Loop 
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Current  and  Potential  Future  Activities 


CURRENT 

1 

FUTURE 

•  Community  Information  Exchange  -  Government/Industry/Academia 

-  National  Symposium  (with  IRIS) 

-  Workshops 

-  International  Symposium 

-  Professional  Journal 

-  Education  Programs 

•  Standards  -  Acquisition,  Test  and  Evaluation 

-  Functional  Model  and  Taxonomy 

-  Engineering  Guidelines 

-  Algorithm/Software  Library 

-Test  Sets 

-  Interfaces 

-  Performance  Metrics 

Evolve  toward  an  Information  Analysis  Center  (lAC) 
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Develop  Strategy  to  Coordinate  Roles 

and  Responsibilities 

Define  Functional  Needs 

-  Standards  for  Acquisition,  System  Engineering,  T&E 

-  Data  exchange  and  commonality 

-  Technology  Development 

-  Information  Exchange 

-  Symposia,  Workshops,  Education,  Publication 

-  Data  Bases  and  Models 


-  Algorithms,  Multi-Source  Test  Sets  and  Models 

Coordinate  with 


-  DUSDA&T 

-  ASD/C^I 

-  DDR&E 

-  CMS 

-  Joint  Staff  (JCS) 

Work  Toward  a  Prototype  I  AC 


DISA  (Dll/COE) 

DARPA 

Labs 

Intel  Agencies 
Services 


Burgeoning  Symposia 
-  Need  Coordination  ? 


•  IEEE  -  Fusion  98  (International  -  Open) 

•  National  Correlation  Working  Group  (NCWG) 

•  ISCAS  -  Special  Session 

•  Exploitation  Technoiogy  Symposium  (ETS) 

•  SPIE  -  Fusion  Sessions 

•  Others? 


AS/FW  3/3/98-  12 


Outline 


•  The  Sensor/Data  Fusion  Working  Group  -  FE  White 

-  Origins 

-  Status  &  Problems 

-  Community  Ideas  and  Feedback 

^  •  The  JDL  Data  Fusion  Model  -  AN  Steinberg 

-  Functional  Model 

-  Taxonomies 

-  Proposed  Revisions 

-  Example  Applications 

-  Summary  Assessment  and  Key  Problems 
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Definitions'll 

A  process  dealing  with  the  association,  correlation,  and  combination  of  data  and  information 
from  single  and  multiple  sources  to  achieve  refined  position  and  identity  estimates,  and 
complete  and  timely  assessments  of  situation  and  threats,  and  their  significance. 

The  process  is  characterized  by  continuous  refinements  of  its  estimates  and  assessments,  and 
by  evaluation  of  of  the  process  itself,  to  achieve  improved  results. 

Sensor  Fusion  =  Data  Fusion  from  Muitipie  Sensors  (same  or  different  sensor 
types) 


Data  Fusion  =  Combining  information  to  estimate  or  predict  the  state  of 
some  aspect  of  the  worid  ^ 


Data  Fusion  Functions: 

-  Data  Alignment 

(spatio-temporal, 
data  normalization, 
evidence  conditioning) 

-  Data  Association 

(hypothesize  entities) 

-  State  Estimation 
&  Prediction 


Signal 


(etc.) 


Situation 

Cross-Force  Relations 
Force  Structure 


(etc.) 


Reports 
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Data  Fusion  Model 

(Revised  1992) 
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JDL  Data  Fusion  Functional  Model 

(Revised  1997) 


Preprocessing/ 
Predetection 
Fusion 
Level  0 


Single  Object 
Refinement 
Level  1 


Situation 
Refinement 
Level  2 


Implications/ 
Threat 
Refinement 
Level  3 


Location;  Behavior;  Aggregate  Situation  Intent;  Courses  of 

Attributes  Class;  ID  object  interpretation  Vulnerability  Action 

refinement 

Database  Services 

Relatively  static  Dynamic 

a  priori  Situation 

Knowledge  Database 
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Definition  of  Component  Functions 


ALIGNMENT: 

CORRELATION: 

ASSOCIATION: 

TRACKING: 

CLASSIFICATION 

IDENTIFICATION 

SITUATION 

ASSESSMENT 

IMPACT 

ASSESSMENT: 

THREAT 

ASSESSMENT: 


Transforming  data  to  common  formats  and  frames  of  reference  (spatio- 
temporal,  measurement  and  attributive)  and  confidence. 

Computing  the  “similarity”  in  data 

Determining  likelihood  that  data  could  have  been  caused  by  the  same  entity 

Creating  and  updating  entity  state  estimates  by  associating  sensor 
measurements  over  time  using  a  predictive  model  of  entity  dynamics 

Determining  that  a  perceived  entity  is  of  a  generic  or  specific  type 

Recognizing  a  perceived  entity  as  particular  individual 

Inferring  relations  among  entities  ,  to  include  force  structure  and  cross  force 
relations,  communications,  physical  context,  etc. 

Estimating  the  significance  (i.e  cost/benefit)  of  perceived  or  predicted 
situations  and  impact  on  planned  actions 

Significance  assessment  regarding  detected  or  predicted  hostile  situations, 
to  include  force  susceptibilities  and  vulnerabilities  to  enemy  postures, 
intentions  and  courses  of  action 
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Data  Fusion  Functional 


The  JDL  model  (1987-91)  and  the  draft  revised  model  (1998) 


•  Level  0  —  Sub-Object  Data  Association  and  Estimation:  pixel/signal  level 
data  association  and  characterization 

•  Level  1 — Object  Refinement:  observation-to-track  association, 
continuous  state  estimation  (e.g.  kinematics)  and  discrete  state  estimation 
(e.g.  target  type  and  ID)  and  prediction 

•  Level  2  —  Situation  Refinement:  object  clustering  and  relational  analysis, 
to  include  force  structure  and  cross  force  relations,  communications, 
physical  context,  etc. 

•  Level  3  —  Impact  Assessment  Threat  Refinement:  intent  estimation, 
ovont  prediction,  consequence  prediction,  susceptibility  and  vulnerability 
assessment 

•  Level  4:  Process  Refinement:  adaptive  data  acquisition  and  processing 
(an  element  of  Resource  Management) 
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Additional  and  Revised  Taxonomies 


•  Physical  Object 

-  Object,  Entity,  Target 

•  System  Representation  of  Physical  Object 

-  Track,  Perceived  Entity,  (Target)  Hypothesis 

•  Object  State  Dimensionality 

-  Attributive/Kinematic,  Discrete/Continuous 

•  Concrete  Physical  Object 

-  Object,  Target,  Platform 

•  Aggregations 

-  Clusters,  Aggregate  Objects,  Ensembles 

•  Data  Alignment 

-  Common  Referencing,  Data  Pre-Processing 
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Data  Fusion  Levels: 
Alternative  Schemes 


Data 

Fusion 

“Level” 

Type  of  Data  Exploited 

Type  of  Process 

Association  |  Estimation 

Type  of  Entity 
Characterized 

0 

Measurements 

•  Pixels 

•  Predetected 

Signals 

•  Detection 

•  Segmentation 

•  Feature  Extraction 

•  Sorting 

•  Feature  State  Est. 

•  Signal  State  Est. 

•  State  Prediction 

•  Region 

•  Feature 

•  (Detected) 

Signal 

1 

•  Regions/  Features 

•  Signals 

•  T  racks 

•  Report/Track 

•  Track/Track 

•  Object  State  Est. 

•  Object  State 
Prediction 

•  Physical  Object 
(Platform,  Target, 
Equipment) 

2 

Relationships 

•  Subordination 

•  Coordination 

•  Conflict 

Relational 

•  Bayesian  Nets 

•  Clustering 

•  Templating 

•  Aggregate  State 

Est. 

•  State  Prediction 

•  Aggregation 

(Unit,  Force  Structure, 
Comm  Net,  etc.) 

•  Situation 

3 

•  Track  State/Prediction 

•  Relationship  to  Own 
Assets 

•  Cost  Model 

•  Filtering 

*  Relational  Analysis 

•  State  Prediction 

•  Cost/  Significance 
Est. 

•  Threat  State  (Event, 
Time,  Relationship) 

•  Event  Prediction 

•  Significance 

4 

•  Object/Situation  State 

•  Event  Prediction/ 
Significance 

•  Response  Plan 
Uncertainty  State 

•  Operations 

Research 

•  Optimal 

Control 

Planning  &  Control 

•  Of  Fusion  Process 

•  Of  Information 
Acquisition 

Process 

Plan  &  Controls 

*  Data  Fusion 

*  Collection  Mgmt 

*  Part  of  Resource  Mgmt 
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SWC  Project  Correlation 
Engineering  Guidelines 


Data  Fusion 
Engineering 
Guideiines 

•  Architectures 

•  Data  Alignment 

•  Data  Correlation 

•  State  Estimation 


•  Svstem  Paradiam 

•  Enaineerina  Process 

•  Taxonomy 

•  Fusion  System  Role  &  Requirements 

•  Evaluation  Metrics: 

•  Functional  Partitioning: 

MOPs,  MOEs 

•  Fusion  Tree 

V 

•  Object-Oriented 

System  Representation 

•  Technique  Applicability: 

Fusion  Nodes  > 
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Data  Fusion  System  Engineering 
Phases  &  Productsi'^^^^^i 


DESIGN  PHASE 

1 .  Operational  Architecture 
Design: 

y  Fusion  Role _ 

2.  System  Architecture 
Design: 

Fusion 
T  Tree 


REQUIREMENTS  REFINEMENT 


CONORS 


Black 

Box 

Spec 


DESIGN 


T 

N 

Sensors/ 

Data 

S 

E 

V 

I 

Sources 

— • 

Fusion 

R 

0 

N 

I 

N 

T 

M 

E 

Response 

,  L 

Resource 

_ 

R 

N 

i 

Systems 

/anaqeme 

It 

A 

MATRIX  A 
SYSTEM  INPUTS 

Q  A  T 


MATRIX  B 

SYSTEM  OUTPUTS/RQMTS 

Q  A  T  Implement 


MATRIX  C 

FUSION  TREE  CHARACTERIZATION 

Tree  Specs  Node  Specs 


3.  Component  Function 
Design: 

A  Fusion  Node 
T  Processing 


MATRIX  D 


NODE  INPUTS 

Q  A  T 


MATRIX  E 

NODE  OUTPUTS/RQMTS 

Q  A  T  Implement 


Source 

d 

e 

g 


MATRIX  F 

FUSION  NODE  CHARACTERIZATION 

Data  Align  Assoc  State  Est. 

Node 
a 
b 
c 


4.  Detailed  Design/ 
Development: 

Pattern  Appiication 


Code 


Note:  Q  A  T  =  Data  Quality,  Availability,  Timeliness 
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Data  Fusion  Design  in  C4iSR 
Architecture  Frameworks 

Matrices  A-B 

~  ~  j  Processing  &  Info 

Operational  Exchange  Requirements 

Identifies  Warfighter  |\j0^  Technological 

_ Needs _  Capabilities 


Processino  and 

Timp-Phq.QpH 

Information 

Systems  Technical 

Exchange 

-  Guidance 

Requirements 

Overlays  Capabilities 

and  Requirements  to 

Identified  Standards 

Matrices  C-F 

Data  Fusion 
Engineering  Guideiines 

Technical 

Identifies  Standards 
and  Conventions 
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Role  of  Data  Fusion  in  an  Integrated 
Information  Processing  System 


Sensors/ 

Sources 

Offensive 


E 

N 

V 

1 

—> 
—> 
— > 

•  Defensive 

•  Nav,  WX 

•  BIT/Cal 

•  Other  Sources 

•  Prior  Models 

R 

3 

0 

N 

M 

Response 

E 

d - 

Systems 

N 

^ - 

•Offensive 

T 

^ - 

•  Defensive 

•Sensor  Contrl 

•  Proc  Control 

•Maintenance/ 

Reconfigure 

•Model  Refinemt 

Data 

Fusion 


Target 
Est/Pred 

Sensor  State 
>  Est/Pred 


Resource 

Management 


Alignment 

State 

Est/Pred 


Own  Force 
State 
Est/Pred 

Model  State 
Est/Pred 
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Z)  C/D  LJJ  QC 


Sensor  Fusion  Performance  is  Affected 

by  System  Context 


Alignment 
Biases 
&  Noise 

Meas 

Error 

State  Est. 
Error 


Hyp 

Gen 


Track  Impurity 
(Tgts/Track) 
Track  Frag 

(Tracks/Tgts) _ ^ 

Hyp  Proliferation 
(Tracks/Report) 


Hyp 
Eva  I 


Hypothesis 

Pruning 

Track  Impurity  I 
(Tgts/Track)  State 

Track  Frag  "►  Est/ 
(Tracks/Tgt)  Predict 

False  Tracks  - 


•  Prior  Predictive  Errors: 
Target  Attributes, 
Behavior,  Densities 
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Fusion  Node  Paradigm 


DATA  FUSION  NODE 


•  DETECT  AMD  RESOLVE 
DATACONFUCTS 

•  CONVERT  DATA  TO 
COI\/l\/ION'ni\/EAMD 
COORIDINATE  FRAI\/E 

•  COI\/PENSATE  FOR 
SOURCE 
l\/ISAUGM\/ENr 


•  GENERATE  FEASIBLE  & 
CONRFaUED  ASSOaATION 
HYFOn-ESES 

•  SCORE  HYPOTHESIZED  DATA 
ASSOaATIONS 

•  SELECT,  DELETE,  OR 
FEEIDBACKDATA 
ASSOaATIONS 


•  ESTIMATHFTTEDICT 
ENTITY  STATES 

-  KaNEMATICS,  ATTRIBUTES, 
ID,  RELTVnONAL  STATES 

•  ESTIMATE  SENSOR/SOURCE 
l\/ISAUGM\/ENTS 

•  FEED  FORWARD  SOURCE/ 
SENSOR  STATUS 
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Alternative  Fusion  Tree  Partitioning 

Schemes^^^^'^i 
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Battlefield  Situation  Awareness  Example 


Visible 


Measurements 

(Images) 


Features 

Decisions 


Features 

Decisions 

FN  — 


•  Measurements 

Received  lntensity[RF/Wa  velength/Time/Angle] 

•  Features 

Size,  Shape,  Orientation,  Angie,  Angie  Rate, 
Range,  Range  Rate 

•  Decisions 

Target  State  and  Uncertainty: 
iocation,  kinematics,  covariance  matrix, 
<p(type  1),  p(type  2),  ...  ,  p(type  N)> 


Decisions 
Level  1 
_ Decision 

FN  h - ► 


Level  2  Level  3 
Decision  Decision 

- ►!  FN  h— ► 


Level  4 
Decision 
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Hybrid  Architecture  Exampie: 

Data  Fusion  and  Resource  Management  Treesi^^ 


RADAR 


OFF-BOARD 
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Summary  Data  Fusion  Technology 
Assessment  (1  of 


DATA  FUSION 
LEVEL 

SUMMARY  OF  THE 
STATE  OF  THE  ART 

CURRENT  LIMITATIONS 

DESIRED  NEAR  TERM 
CAPABILITIES 

Level  1 : 

Positional, 

Kinematic, 

Attribute 

Estimation 

•  Relatively  mature 

-  numerous  techniques 
for  tracking 

-  current  research 

in  MHT,  JPDA  trackers 

•  Object  I/D  fusion  dominated 
by  feature  &  decision 
methods 

-  current  R&D  in  ANS  and 
syntactic  methods 

•  Difficulty  tracking  targets  in 
dense  target  environment,  low 
SNR,  maneuvering  targets 

•  Selection  of  attributes  for 
classification 

•  Selection/use  of  multiple 
techniques  in  concert 

•  Off-the-shelf  software 
package  for  robust 
estimation 

•  Multi-technique  approach 
for  object  I/D 

•  Methodology  &  guidelines 
for  algorithm  selection 

•  Standard  test  beds,  data 
sets 

Levels  2  and  3: 
Situation  and 
Significance 

•  Relatively  immature 

-  heuristic  techniques 
include  templating, 
expert  systems 

-  numerous  experimental 
prototypes 

•  Doctrinal  Basis  not  well-defined 

•  Limited  understanding  of 
decision  makers  needs 

•  Evolving  threat  environment 

•  Limited  cognitive  models 

•  Robust  techniques  to 
solve  subset  of 
situation/threat 
refinement 

•  Basis  for  cognitive 
models 

Level  4: 

Process 

Refinement 

•  Mixed  maturity 

-  well  founded  technology 
for  single  sensor 

-  immature  for  multi-sensors 

-  MOPs  defined 

-  prototype  expert  systems 

•  MOE  not  well-defined 

•  Disconnect  between  mission 
management  and  fusion 
management 

•  Hybrid  architectures 
challenging 

•  MOE/MOP  Consensus 

•  Metrics  baseline 

•  Generic  architecture  and 
techniques  for  multi¬ 
sensor  control 
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Summary  Data  Fusion  Technology 
Assessment  (2  of  2)m 


DATA  FUSION 
FUNCTIONAL 
AREA 

SUMMARY  OF  THE 

STATE  OF  THE  ART 

CURRENT  LIMITATIONS 

DESIRED  NEAR  TERM 
CAPABILITIES 

Human- 

Computer 

interface 

(HCI) 

•  Numerous  tools  for  rapid 
prototyping 

•  Current  research  in  display 
design,  crew  position  layout, 
workload  aspects 

•  Ergonomic  vice  cognitive  focus 

•  Limited  HCI  research  specific  to 
data  fusion 

•  Limited  cognitive  models  for 
focus  of  attention,  stress 
management,  alternative 
decision  styles 

•  Integrated  exploitation  of 
advanced  technology  (e.g., 
HDTV,  virtual  reality,  multi- 
media) 

•  Intelligent  Groupware 

•  Multi-person  HCI 

Data  Base 
Management 

•  Numerous  commercial  tools 
(relational  models) 

•  Fourth-generation  query 
languages 

•  Trend  toward  object-oriented 
DBMS 

•  Simultaneous  optimization  of 
storage  and  retrieval 

•  Distributed  concurrency 

•  Multi-level  security 

•  Natural  language  interfaces 

•  S/W  based  solution  to  multi¬ 
level  security 

•  COTS  DBMS  to  handle 
diverse  data  (image,  text, 
data,  KBS) 

Development 

Environment 

•  Robust  development  standards 
and  procedures  for  conventional 
systems 

•  Widespread  development  of 
application  specific  prototypes 

•  Single  vs.  multi-sensor  models 

•  Lack  of  Standard  MOPs  and 
test  sets 

•  Disjoint  test  beds  and  simulation 
tools 

•  Limited  tools/MOE  for  Level  2,3 
fusion 

•  Robust  test-bed  for  Test  and 
Evaluation 

•  Metrics  for  MOP/MOE 

•  Fusion  Software  Library  and 
Clearinghouse 

•  Data  Fusion  System 
Engineering  methodology 
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Community  Update  -  Summary 


•  Pervasiveness  of  Data  Fusion  Is  a  Double-edged  Sword: 

-  Belongs  to  Everybody  but  Nobody 

•  Infrastructure  is  Needed  to  Reduce  Acquisition,  Development 
and  Operational  Costs 

-  Architecture  Standards 

-  Coordination 

-  DOD,  Other  Government,  Industry,  Academia,  Internationa 

•  Community  Needs  Activism 

-  Next  Generation  of  Leaders 
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